fix(compilers/openapi)!: scan for the version key past the cap - #441
Conversation
…for it Detection parsed whole documents to find one key. sniffWhole ran yaml.Unmarshal over everything past the 64 KiB cap, before the loader's size and node budgets and with no context to cancel it: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway. main declines that file in 11 ms. #441 scoped the key search but left the parse. The scan the guard already ran is extended to read the value beside the key, which answers the whole question without building a tree: 8 MB 209 ms / 82 MB -> 1.9 ms / 0 allocations 32 MB 805 ms / 333 MB -> 4.1 ms / 0 allocations Reading the value, rather than only finding the key, closes two more holes. Key order stopped deciding the format: the prefix answered on whichever key it reached and returned, so a document declaring both read as swagger@2.0 above the cap and openapi@3.0 below it — one document, two answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat claims. And a key alone is no longer a declaration: prose beside the word (`openapi: is a format` in a Markdown file, at any size) is declined in silence instead of claimed and reported under this compiler's parse error. BREAKING: detection past the cap no longer emits undecodable-source. It cannot: it has read one key, not the document. A source it names and the loader cannot parse is now the compile's finding, so openapi.Compile turns load.ErrParse into that diagnostic rather than returning a Go error. engine.Run wraps a compiler's Go error in its own and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without this a broken spec would be reported as a misuse of morphic. Exit 1 and the diagnostic are preserved, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake. The prefix machinery goes with the parse it existed to avoid — sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap. Tests: the six mutations the review planted all stayed green. Twelve against the new code, including all six where they still apply, are all caught. Adds the engine.Run case over a generated >64 KiB JSON with the version last — nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB — and the cap-boundary cases, where a broken document is what tells the two readings apart. Refs #420 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
…for it Detection parsed whole documents to find one key. sniffWhole ran yaml.Unmarshal over everything past the 64 KiB cap, before the loader's size and node budgets and with no context to cancel it: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway. main declines that file in 11 ms. #441 scoped the key search but left the parse. The scan the guard already ran is extended to read the value beside the key, which answers the whole question without building a tree: 8 MB 209 ms / 82 MB -> 1.9 ms / 0 allocations 32 MB 805 ms / 333 MB -> 4.1 ms / 0 allocations Reading the value, rather than only finding the key, closes two more holes. Key order stopped deciding the format: the prefix answered on whichever key it reached and returned, so a document declaring both read as swagger@2.0 above the cap and openapi@3.0 below it — one document, two answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat claims. And a key alone is no longer a declaration: prose beside the word (`openapi: is a format` in a Markdown file, at any size) is declined in silence instead of claimed and reported under this compiler's parse error. BREAKING: detection past the cap no longer emits undecodable-source. It cannot: it has read one key, not the document. A source it names and the loader cannot parse is now the compile's finding, so openapi.Compile turns load.ErrParse into that diagnostic rather than returning a Go error. engine.Run wraps a compiler's Go error in its own and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without this a broken spec would be reported as a misuse of morphic. Exit 1 and the diagnostic are preserved, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake. The prefix machinery goes with the parse it existed to avoid — sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap. Tests: the six mutations the review planted all stayed green. Twelve against the new code, including all six where they still apply, are all caught. Adds the engine.Run case over a generated >64 KiB JSON with the version last — nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB — and the cap-boundary cases, where a broken document is what tells the two readings apart. Refs #420 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
72addaf to
4322dc7
Compare
…for it Detection parsed whole documents to find one key. sniffWhole ran yaml.Unmarshal over everything past the 64 KiB cap, before the loader's size and node budgets and with no context to cancel it: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway. main declines that file in 11 ms. #441 scoped the key search but left the parse. The scan the guard already ran is extended to read the value beside the key, which answers the whole question without building a tree: 8 MB 209 ms / 82 MB -> 1.9 ms / 0 allocations 32 MB 805 ms / 333 MB -> 4.1 ms / 0 allocations Reading the value, rather than only finding the key, closes two more holes. Key order stopped deciding the format: the prefix answered on whichever key it reached and returned, so a document declaring both read as swagger@2.0 above the cap and openapi@3.0 below it — one document, two answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat claims. And a key alone is no longer a declaration: prose beside the word (`openapi: is a format` in a Markdown file, at any size) is declined in silence instead of claimed and reported under this compiler's parse error. BREAKING: detection past the cap no longer emits undecodable-source. It cannot: it has read one key, not the document. A source it names and the loader cannot parse is now the compile's finding, so openapi.Compile turns load.ErrParse into that diagnostic rather than returning a Go error. engine.Run wraps a compiler's Go error in its own and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without this a broken spec would be reported as a misuse of morphic. Exit 1 and the diagnostic are preserved, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake. The prefix machinery goes with the parse it existed to avoid — sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap. Tests: the six mutations the review planted all stayed green. Twelve against the new code, including all six where they still apply, are all caught. Adds the engine.Run case over a generated >64 KiB JSON with the version last — nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB — and the cap-boundary cases, where a broken document is what tells the two readings apart. Refs #420 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
4322dc7 to
e503898
Compare
…for it Detection parsed whole documents to find one key. sniffWhole ran yaml.Unmarshal over everything past the 64 KiB cap, before the loader's size and node budgets and with no context to cancel it: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway. main declines that file in 11 ms. #441 scoped the key search but left the parse. The scan the guard already ran is extended to read the value beside the key, which answers the whole question without building a tree: 8 MB 209 ms / 82 MB -> 1.9 ms / 0 allocations 32 MB 805 ms / 333 MB -> 4.1 ms / 0 allocations Reading the value, rather than only finding the key, closes two more holes. Key order stopped deciding the format: the prefix answered on whichever key it reached and returned, so a document declaring both read as swagger@2.0 above the cap and openapi@3.0 below it — one document, two answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat claims. And a key alone is no longer a declaration: prose beside the word (`openapi: is a format` in a Markdown file, at any size) is declined in silence instead of claimed and reported under this compiler's parse error. BREAKING: detection past the cap no longer emits undecodable-source. It cannot: it has read one key, not the document. A source it names and the loader cannot parse is now the compile's finding, so openapi.Compile turns load.ErrParse into that diagnostic rather than returning a Go error. engine.Run wraps a compiler's Go error in its own and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without this a broken spec would be reported as a misuse of morphic. Exit 1 and the diagnostic are preserved, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake. The prefix machinery goes with the parse it existed to avoid — sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap. Tests: the six mutations the review planted all stayed green. Twelve against the new code, including all six where they still apply, are all caught. Adds the engine.Run case over a generated >64 KiB JSON with the version last — nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB — and the cap-boundary cases, where a broken document is what tells the two readings apart. Refs #420 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
e503898 to
02d3922
Compare
…for it Detection parsed whole documents to find one key. sniffWhole ran yaml.Unmarshal over everything past the 64 KiB cap, before the loader's size and node budgets and with no context to cancel it: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway. main declines that file in 11 ms. #441 scoped the key search but left the parse. The scan the guard already ran is extended to read the value beside the key, which answers the whole question without building a tree: 8 MB 209 ms / 82 MB -> 1.9 ms / 0 allocations 32 MB 805 ms / 333 MB -> 4.1 ms / 0 allocations Reading the value, rather than only finding the key, closes two more holes. Key order stopped deciding the format: the prefix answered on whichever key it reached and returned, so a document declaring both read as swagger@2.0 above the cap and openapi@3.0 below it — one document, two answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat claims. And a key alone is no longer a declaration: prose beside the word (`openapi: is a format` in a Markdown file, at any size) is declined in silence instead of claimed and reported under this compiler's parse error. BREAKING: detection past the cap no longer emits undecodable-source. It cannot: it has read one key, not the document. A source it names and the loader cannot parse is now the compile's finding, so openapi.Compile turns load.ErrParse into that diagnostic rather than returning a Go error. engine.Run wraps a compiler's Go error in its own and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without this a broken spec would be reported as a misuse of morphic. Exit 1 and the diagnostic are preserved, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake. The prefix machinery goes with the parse it existed to avoid — sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap. Tests: the six mutations the review planted all stayed green. Twelve against the new code, including all six where they still apply, are all caught. Adds the engine.Run case over a generated >64 KiB JSON with the version last — nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB — and the cap-boundary cases, where a broken document is what tells the two readings apart. Refs #420 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
02d3922 to
9a343cb
Compare
Wahbeh-Mohammad
left a comment
There was a problem hiding this comment.
Went through this against binaries built from origin/main, from stack/6-constraints, and from this head, running morphic validate over generated specs sitting on both sides of the 64 KiB cap. Six things below; the inline comments have the detail.
Four of them (BOM, merge key, document boundaries, and the test that claims size-independence) are one story — scanProbe reads a document differently from decodeYAML, and the cap decides which reading answers — so they want one design rather than four patches. The shape I'd suggest:
Fix in the scan, where it claims or declines something the parser disagrees with:
- a leading UTF-8 BOM defeats both arms (a regression against
main); - the block scan crosses YAML document boundaries, so it answers for a document nothing downstream reads;
openapi:3.1.0counts as a key, which YAML says it is not.
Declare, don't chase, where a byte scan genuinely cannot reach:
- a root
<<merge key — following it means resolving an anchor, which means the parse the cap exists to avoid, and the input chooses when to trigger it; - a block document quoting its top-level key (already in the body);
- flow style without quotes,
{openapi: 3.1.0, ...}; - an anchor or a tag before the version,
openapi: &v 3.1.0/openapi: !!str 3.1.0.
The declaration is worth more as a test than as prose. A table that runs both readings over the same bytes, every row either "must agree" or carrying the reason it does not, means adding a tolerance deletes a reason and losing one adds a row — neither can happen silently, and prose in a doc comment can do neither.
I prototyped all of it locally: go test ./... green, compilers/openapi statement coverage still 100.0%, and three planted mutations (drop the BOM trim / drop the document bound / let the colon stand unseparated) each redden only their own cases.
The title lands a breaking change unmarked, and names the smaller half of it
fix(compilers/openapi): scope the key scan, keep the read's error carries no !, while the body opens with Breaking and commit 34ebb32 is fix(compilers/openapi)!:. PRs here squash-merge, so the title is the subject that lands on main — the ! has to be on it. Adding one to the current title does not fit:
$ printf '%s (#441)' "fix(compilers/openapi)!: scope the key scan, keep the read's error" | wc -c
73
It also describes commit 1 rather than the merged change: "scope the key scan" survives, but the headline is that detection past the cap now reads one key instead of parsing the document, and that is the breaking half. Suggestion:
$ printf '%s (#441)' "fix(compilers/openapi)!: scan for the version key past the cap" | wc -c
69
(fix(compilers/openapi)!: read the version key without parsing is 68 if you prefer naming the cost rather than the boundary.)
Two sentences in the body are now falsified
- "every case is at least as good as
main, and four are better" — the BOM cases and the merge-key case are each worse thanmainon this head. Worth re-measuring after the BOM fix and restating with the merge-key loss named. - "One deliberate loss, for your judgement" — there are more than one, per the list above. Since the repo asks that a deliberate limitation be stated in the code and in the body somewhere the next reader reaches, the body wants the full list and
scanProbe's doc wants a pointer to the test that holds it.
Detection parsed whole documents to find one key. sniffWhole ran yaml.Unmarshal over everything past the 64 KiB cap, before the loader's size and node budgets and with no context to cancel it: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway. main declines that file in 11 ms. #441 scoped the key search but left the parse. The scan the guard already ran is extended to read the value beside the key, which answers the whole question without building a tree: 8 MB 209 ms / 82 MB -> 1.9 ms / 0 allocations 32 MB 805 ms / 333 MB -> 4.1 ms / 0 allocations Reading the value, rather than only finding the key, closes two more holes. Key order stopped deciding the format: the prefix answered on whichever key it reached and returned, so a document declaring both read as swagger@2.0 above the cap and openapi@3.0 below it — one document, two answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat claims. And a key alone is no longer a declaration: prose beside the word (`openapi: is a format` in a Markdown file, at any size) is declined in silence instead of claimed and reported under this compiler's parse error. BREAKING: detection past the cap no longer emits undecodable-source. It cannot: it has read one key, not the document. A source it names and the loader cannot parse is now the compile's finding, so openapi.Compile turns load.ErrParse into that diagnostic rather than returning a Go error. engine.Run wraps a compiler's Go error in its own and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without this a broken spec would be reported as a misuse of morphic. Exit 1 and the diagnostic are preserved, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake. The prefix machinery goes with the parse it existed to avoid — sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap. Tests: the six mutations the review planted all stayed green. Twelve against the new code, including all six where they still apply, are all caught. Adds the engine.Run case over a generated >64 KiB JSON with the version last — nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB — and the cap-boundary cases, where a broken document is what tells the two readings apart. Refs #420 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
9a343cb to
42a90f6
Compare
|
Title is now The two falsified sentences are gone. The body's new section lists what the scan fixes and the whole of what it declines, each shape against The "A doc sentence" section went with its commit, which moved down to #437 where the review asked for it. |
42a90f6 to
16cb7ba
Compare
16cb7ba to
d48f3eb
Compare
d48f3eb to
fded2b2
Compare
Detection parsed whole documents to find one key. sniffWhole ran yaml.Unmarshal over everything past the 64 KiB cap, before the loader's size and node budgets and with no context to cancel it: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway. main declines that file in 11 ms. #441 scoped the key search but left the parse. The scan the guard already ran is extended to read the value beside the key, which answers the whole question without building a tree: 8 MB 209 ms / 82 MB -> 1.9 ms / 0 allocations 32 MB 805 ms / 333 MB -> 4.1 ms / 0 allocations Reading the value, rather than only finding the key, closes two more holes. Key order stopped deciding the format: the prefix answered on whichever key it reached and returned, so a document declaring both read as swagger@2.0 above the cap and openapi@3.0 below it — one document, two answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat claims. And a key alone is no longer a declaration: prose beside the word (`openapi: is a format` in a Markdown file, at any size) is declined in silence instead of claimed and reported under this compiler's parse error. BREAKING: detection past the cap no longer emits undecodable-source. It cannot: it has read one key, not the document. A source it names and the loader cannot parse is now the compile's finding, so openapi.Compile turns load.ErrParse into that diagnostic rather than returning a Go error. engine.Run wraps a compiler's Go error in its own and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without this a broken spec would be reported as a misuse of morphic. Exit 1 and the diagnostic are preserved, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake. The prefix machinery goes with the parse it existed to avoid — sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap. Tests: the six mutations the review planted all stayed green. Twelve against the new code, including all six where they still apply, are all caught. Adds the engine.Run case over a generated >64 KiB JSON with the version last — nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB — and the cap-boundary cases, where a broken document is what tells the two readings apart. Refs #420 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
fded2b2 to
3240bc7
Compare
The whole-source read #420 added got two answers wrong, both by reusing machinery written for a cut prefix. The flow decoder reports "this is a flow mapping" for anything opening with `{`, and drops the error that ended its walk. For a cut prefix that is right: the cut always breaks the token stream, so the error describes the cut and not the document. For a whole document it hides the document's own break. A JSON source past the cap whose `openapi` key sits behind a syntax error came back with a nil error, so Detect saw no failure to report and declined it as an unrecognized format — the very answer #420 set out to replace, still standing for every JSON source, which is the style the motivating spec is written in. The decoder now returns the error that stopped it and treats stopping on the mapping's own closing delimiter or on the entry cap as no error at all; sniffPrefix drops it along with the cut that caused it, and sniffWhole keeps it. The key scan was widened to the whole source without being scoped to the top level. Its block arm reads column 0 and always was top-level, but its quoted arm matched `"openapi":` at any depth, anywhere in the buffer. Bounded to the first 64 KiB that cost a needless parse; over a whole source it makes a claim, and a wrong one — another format's document that nests such a key and does not parse was reported as an undecodable OpenAPI source. Saying nothing about bytes that are not this compiler's own is the rule detection is built on. The quoted spelling is how flow style writes every key, so flow structure is what scopes it: a depth-tracking scan reads the root mapping's own entries and nothing under them, and a source that opens no mapping at all declares nothing here. It is a lexer rather than a parser because the case it exists for is a document broken before the key that names it, where there is no tree to ask. A block document that quotes its top-level key is no longer seen and is declined in silence, which is the direction to be wrong in. A valid document past the cap that declares its version last still compiles, which is what #420 was about. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SYeBgDsskwnyitLgPGCPn1
Detection parsed whole documents to find one key. sniffWhole ran yaml.Unmarshal over everything past the 64 KiB cap, before the loader's size and node budgets and with no context to cancel it: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway. main declines that file in 11 ms. #441 scoped the key search but left the parse. The scan the guard already ran is extended to read the value beside the key, which answers the whole question without building a tree: 8 MB 209 ms / 82 MB -> 1.9 ms / 0 allocations 32 MB 805 ms / 333 MB -> 4.1 ms / 0 allocations Reading the value, rather than only finding the key, closes two more holes. Key order stopped deciding the format: the prefix answered on whichever key it reached and returned, so a document declaring both read as swagger@2.0 above the cap and openapi@3.0 below it — one document, two answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat claims. And a key alone is no longer a declaration: prose beside the word (`openapi: is a format` in a Markdown file, at any size) is declined in silence instead of claimed and reported under this compiler's parse error. BREAKING: detection past the cap no longer emits undecodable-source. It cannot: it has read one key, not the document. A source it names and the loader cannot parse is now the compile's finding, so openapi.Compile turns load.ErrParse into that diagnostic rather than returning a Go error. engine.Run wraps a compiler's Go error in its own and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without this a broken spec would be reported as a misuse of morphic. Exit 1 and the diagnostic are preserved, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake. The prefix machinery goes with the parse it existed to avoid — sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap. Tests: the six mutations the review planted all stayed green. Twelve against the new code, including all six where they still apply, are all caught. Adds the engine.Run case over a generated >64 KiB JSON with the version last — nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB — and the cap-boundary cases, where a broken document is what tells the two readings apart. Refs #420 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
The commit that replaced the past-the-cap parse with a scan left maxSniffBytes' comment naming sniffWhole, which the same commit deleted, and describing a document past the cap as "read whole" when it is now not read as a tree at all. Says what the constant does: it is where detection stops parsing and scans, and nothing is declined for being large. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016EHKV7ZYQJJXCPyynTWq4P
The scan and the parse read a document differently, and the cap decides which answers, so every shape they disagree on names one format below 64 KiB and another above it. Three were the scan's to fix: a UTF-8 byte-order mark defeated both arms, so a spec an editor had marked compiled at the cap and was unrecognized one byte past it — the #420 failure back for a new input class; the block scan crossed YAML document boundaries, claiming a key in a document the compile never parses; and it took `openapi:3.1.0`, a plain scalar, for a key the parser says declares nothing. The mark is trimmed, both arms are bounded to the first document with the opening marker left behind so a flow document after `---` reads as one, and a block entry needs the separated colon YAML needs. The shapes the scan does not read by design — a root merge key, whose `<<` means resolving an anchor and so the parse the cap exists to avoid, on an input the source chooses; a quoted key in block style; an unquoted one in flow style; an anchor or a tag before the version — are declared in a differential table that runs both readings over one set of bytes, each declared row against its reason and asserted in both directions, so a tolerance added deletes a reason and a shape lost adds a row. declaresProbeKey is reached only after a parse failed, which only runs at or below the cap, so its doc described a call site this branch deleted and the test feeding it input past the cap exercised bytes it cannot be handed; the one row that was not a duplicate moves to the scoping test and the rest go. The key-order test is named for the one shape it pins rather than for a size-independence the branch does not have. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T5no6iADeMGgYjsYcV5in
undecodable said the source table exists once a parse has failed. It does not: the parse that failed is the one that would have built the document, Compile returns none, and a Source of 0 against the nil document engine.Run hands on resolves to no path — naming nothing while claiming to. The loader's own message carries the position, so the diagnostic is sited at NoSource, as Detect's sibling for the same condition already is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011T5no6iADeMGgYjsYcV5in
3240bc7 to
82acc52
Compare
Breaking — see the last section. Fixes the defects two adversarial reviews of the whole stack found in stack PR 1, which the per-commit gates could not see.
#420's fix missed malformed JSON
decodeFlowEntriesreturned(probe, true)whenever the input opened with{, swallowing a mid-stream JSON error. That contract is correct for a cut prefix (the cut always errors) and wrong for a whole document, which the newsniffWholereuses it for. So a >64 KiB JSON document whoseopenapikey sat behind a syntax error still reportedunrecognized-format— the outcome PR 1 says it replaced.Correction to an earlier version of this description, from the review on PR 1: this was never the motivating case. Valid JSON — Stripe's
spec3.json— worked in PR 1 from the start: the cut prefix yields an empty probe, andsniffWholethen reads the whole document successfully. Only malformed JSON was missed.The key scan was not top-level-scoped
"openapi":matched at any nesting depth, anywhere in the buffer. Onmainthat was bounded to 64 KiB; PR 1 removed the bound without tightening the predicate. So a foreign document nesting an"openapi"key paid a full parse, and one that also failed to parse could be reported asopenapi/undecodable-source— this compiler claiming bytes that are not its own, whichdetect.go's own comments forbid.declaresProbeKeyis nowdeclaresBlockKey(bare name at column 0, unchanged) ORdeclaresFlowKey, the latter requiring the source to open a flow mapping and matching only at depth 1. The 64 KiB bound was not reinstated.Both fixes were mutation-checked: swallowing the error again reddens the new flow test; restoring the unscoped scan reddens 7 subtests. How the scan compares with
mainand with the parse, shape by shape, is the section below.Detection parsed the whole document to find one key
sniffWholeranyaml.Unmarshalover everything past the cap, ahead of the loader's size and node budgets and with no context to cancel it. Measured insideDetect: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway;maindeclines it in 11 ms. The earlier commit on this branch scoped the key search but deliberately did not reinstate a bound.The scan the guard already ran now reads the value beside the key, which answers the whole question without building a tree:
Reading the value closes two more holes the review found:
{"swagger":"2.0", …64 KiB…, "openapi":"3.0.3"}read asswagger@2.0above the cap andopenapi@3.0below it — one document, two answers, which is exactly the propertyTestDetect_KeyOrderDoesNotDecideTheFormatclaims to pin.openapi: is a formatat column 0 in a Markdown file, at any size — is declined in silence instead of claimed and reported under this compiler's parse error. That also fixes the same shape below the cap, which predates the stack:mainreports such a file asopenapi@is a format.BREAKING. Detection past the cap no longer emits
undecodable-source; it cannot, having read one key rather than the document. A source it names and the loader cannot parse is now the compile's finding, soopenapi.Compileturnsload.ErrParseinto that diagnostic instead of returning a Go error. That conversion is not cosmetic:engine.Runwraps a compiler's Go error in its own, and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without it a broken spec would be reported as a misuse ofmorphic. Exit 1 and the diagnostic are preserved end-to-end, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake.The prefix machinery goes with the parse it existed to avoid —
sniffPrefix,sniffWhole,decodeFlowEntries,wholeLinesand the entry cap.On the review's test finding. All six mutations the review planted stayed green on this branch. Twelve against the new code — including all six where they still apply — are all caught. Adds the
engine.Runcase the review asked for, over a generated >64 KiB JSON with the version last: nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB. Adds the cap-boundary cases too, where a broken document is what tells the two readings apart — a document both can read cannot, which is why an off-by-one there survived until now.What the scan reads, and what it declines
The scan reads less than the parse, and the cap decides which answers, so every shape they disagree on is a document that names one format below 64 KiB and another above it.
TestReadings_AgreeExceptWhereDeclaredruns both readings over one set of bytes: every row either must agree or carries the reason it does not, asserted in both directions, so a tolerance added to the scan deletes a reason and a shape lost adds a row. The committed corpus cannot see any of this — its largest spec is a few KiB — which is why the table constructs the shapes.Fixed in the scan, where it claimed or declined something the parser disagrees with (each a regression against
mainor a false claim, found in review):unrecognized-formatone byte past it;openapi:3.1.0at column 0 counted as a key; YAML reads it as a plain scalar, and the parse refuses that document.Declared, not chased — the whole list of shapes the scan reads less than the parse, each held to its reason by the table:
<<merge key: following it means resolving an anchor, which means the parse the cap exists to avoid, and the input chooses when to trigger it.mainread this shape past the cap; this branch declines it in silence."openapi": 3.0.0at column 0): admitting it re-opens the guard for zero-indent JSON, whichjson.MarshalIndent(v, "", "")really produces. Declined onmaintoo.{openapi: 3.1.0, …}: the flow scan is a JSON lexer. Declined onmaintoo.openapi: &v 3.1.0/openapi: !!str 3.1.0: the scan reads the scalar as written, which is no version. Declined onmaintoo.openapi:3.1.0past the cap goes fromundecodable-sourceonmainto a silent decline — the same class as theopenapi: [unterminatedloss the breaking section describes, listed here so it is in one place.Stack 7 of 8. Base
stack/6-constraints— review and merge bottom-up. Every commit here passedmake gatewhen it landed, and the full gate was re-run on the top of the stack. Run it asGOTOOLCHAIN=go1.26.3 make gate; this machine's Go 1.27 fails it for reasons unrelated to any change (#431).🤖 Generated with Claude Code
https://claude.ai/code/session_01SYeBgDsskwnyitLgPGCPn1